Deep Sparse Rectifier Neural Networks
نویسندگان
چکیده
Rectifying neurons are more biologically plausible than logistic sigmoid neurons, which are themselves more biologically plausible than hyperbolic tangent neurons. However, the latter work better for training multi-layer neural networks than logistic sigmoid neurons. This paper shows that networks of rectifying neurons yield equal or better performance than hyperbolic tangent networks in spite of the hard non-linearity and non-differentiability at zero and create sparse representations with true zeros which are remarkably suitable for naturally sparse data. Even though they can take advantage of semi-supervised setups with extraunlabeled data, deep rectifier networks can reach their best performance without requiring any unsupervised pre-training on purely supervised tasks with large labeled datasets. Hence, these results can be seen as a new milestone in the attempts at understanding the difficulty in training deep but purely supervised neural networks, and closing the performance gap between neural networks learnt with and without unsupervised pre-training.
منابع مشابه
Document Classification with Deep Rectifier Neural Networks and Probabilistic Sampling
Deep learning is regarded by some as one of the most important technological breakthroughs of this decade. In recent years it has been shown that using rectified neurons, one can match or surpass the performance achieved using hyperbolic tangent or sigmoid neurons, especially in deep networks. With rectified neurons we can readily create sparse representations, which seems especially suitable f...
متن کاملSpeeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks. An ReLU converts negative values to zeros, and does not change positive values, which leads to a high sparsity of neurons. In this work, we first examine the sparsity of the outputs of ReLUs in some popular deep convolutional architectures. And then we use the sparsity property of ReLUs to accelerate the calcul...
متن کاملNet-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee
Model reduction is a highly desirable process for deep neural networks. While large networks are theoretically capable of learning arbitrarily complex models, overfitting and model redundancy negatively affects the prediction accuracy and model variance. NetTrim is a layer-wise convex framework to prune (sparsify) deep neural networks. The method is applicable to neural networks operating with ...
متن کاملNet-Trim: A Layer-wise Convex Pruning of Deep Neural Networks
and quantum settings Model reduction is a highly desirable process for deep neural networks. While large networks are theoretically capable of learning arbitrarily complex models, overfitting and model redundancy negatively affects the prediction accuracy and model variance. Net-Trim is a layer-wise convex framework to prune (sparsify) deep neural networks. The method is applicable to neural ne...
متن کاملPosterior Concentration for Sparse Deep Learning
Spike-and-Slab Deep Learning (SS-DL) is a fully Bayesian alternative to Dropout for improving generalizability of deep ReLU networks. This new type of regularization enables provable recovery of smooth input-output maps with unknown levels of smoothness. Indeed, we show that the posterior distribution concentrates at the near minimax rate for α-Hölder smooth maps, performing as well as if we kn...
متن کامل